.TH funcslower 8  "2017-03-30" "USER COMMANDS"
.SH NAME
funcslower \- Trace slow kernel or user function calls.
.SH SYNOPSIS
.B funcslower [\-hf] [\-p PID] [\-U | \-K] [-m MIN_MS] [-u MIN_US] [-a ARGUMENTS] [-T] [-t] [-v] function [function ...]
.SH DESCRIPTION
This script traces a kernel or user function's entry and return points, and
prints a message when the function's latency exceeded the specified threshold.
Multiple functions are supported, and you can mix kernel functions with user
functions in different libraries.

WARNING: See the OVERHEAD section.

By default, a minimum millisecond threshold of 1 is used. Recursive functions
are not supported: only the inner-most recursive invocation will be traced.

Since this uses BPF, only the root user can use this tool.
.SH REQUIREMENTS
CONFIG_BPF and bcc.
.SH OPTIONS
\-p PID
Trace this PID only.
.TP
\-m MIN_NS
Minimum duration to trace, in milliseconds. Default is 1 ms.
.TP
\-u MIN_US
Minimum duration to trace, in microseconds.
.TP
\-a ARGUMENTS
Print the function's arguments, up to 6.
.TP
\-T
Print a HH:MM:SS timestamp with each entry.
.TP
\-t
Print a seconds timestamp with each entry, at microsecond resolution.
.TP
\-f
Print output in folded stack format.
.TP
\-U
Show stacks from user space only (no kernel space stacks).
.TP
\-K
Show stacks from kernel space only (no user space stacks).
.TP
\-v
Print the resulting BPF program, for debugging purposes.
.TP
function
The function to trace -- multiple functions are supported. If a plain function
name is provided, the function is assumed to be a kernel function. For user
functions, provide the library name and the function name, e.g. bash:readline
or c:malloc.
.SH EXAMPLES
.TP
Trace vfs_write calls slower than 1ms:
#
.B funcslower vfs_write
.TP
Trace open() calls in libc slower than 10us:
#
.B funcslower \-u 10 c:open
.TP
Trace both malloc() and free() slower than 10us, in pid 135 only:
#
.B funcslower \-p 135 \-u 10 c:malloc c:free
.TP
Trace the write syscall and print its first 4 arguments:
#
.B funcslower -a 4 SyS_write
.TP
Trace opens from libc and print the user and kernel stack frames:
#
.B funcslower -UK c:open
.SH FIELDS
.TP
TIME
Time of the event as a human-readable HH:MM:SS format, or a timestamp in seconds
at microsecond-accuracy from the first event seen.
.TP
COMM
Process name.
.TP
PID
Process ID.
.TP
LAT
Latency of the operation in either microseconds (us) or milliseconds (ms).
.TP
RVAL
The return value from the function. Often useful for diagnosing a relationship
between slow and failed function calls.
.TP
FUNC
The function name, followed by its arguments if requested.
.SH OVERHEAD
Depending on the function(s) being traced, overhead can become severe. For
example, tracing a common function like malloc() can slow down a C/C++ program
by a factor of 2 or more. On the other hand, tracing a low-frequency event like
the SyS_setreuid() function will probably not be as prohibitive, and in fact
negligible for functions that are called up to 100-1000 times per second.

You should first use the funclatency and argdist tools for investigation,
because they summarize data in-kernel and have a much lower overhead than this
tool. To get a general idea of the number of times a particular function is
called (and estimate the overhead), use the funccount tool, e.g.:
.PP
# funccount c:open
.SH SOURCE
This is from bcc.
.IP
https://github.com/iovisor/bcc
.PP
Also look in the bcc distribution for a companion _examples.txt file containing
example usage, output, and commentary for this tool.
.SH OS
Linux
.SH STABILITY
Unstable - in development.
.SH AUTHOR
Sasha Goldshtein
.SH SEE ALSO
funccount(8), funclatency(8), argdist(8), trace(8)
